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1.  SUMMARY  OF  RESEARCH  PROGRESS  AND  RESULTS 

During  the  year  supported  by  this  grant,  we  have  made  significant  progress  both 
in  areas  we  proposed  to  investigate  and  in  related  areas.  Ir  this  section,  we  sum¬ 
marize  the  progress  in  those  areas  that  have  resulted  in  publications. 

1.1.  Stochastic  Control. 

1.1.1.  Stochastic  Control  of  Markov  Processes. 

We  have  continued  our  research  program  in  a  major  new  area  involving  adaptive 
estimation  and  control  problems  for  stochastic  systems  involving  either  incomplete 
(or  noisy)  observations  of  the  state  or  nonlinear  dynamics.  The  first  class  of  prob¬ 
lems  we  have  been  studying  involves  finite  state  Markov  chains  with  incomplete 
state  observations  and  unknown  parameters;  in  particular,  we  have  studied  certain 
classes  of  quality  control,  replacement,  and  repair  problems. 

We  have  first  considered  a  quality  control  problem  in  which  a  system,  such  as  a 
manufacturing  unit  or  computer  communications  network,  can  be  in  either  of  two 
states:  good  or  bad.  Control  actions  available  to  the  inspect  or/ decision-maker  are: 

(a)  produce  without  inspection, 

(b)  produce  and  inspect;  or 

(c)  repair. 

Under  actions  (a)  and  (b)  the  system  is  subject  to  Markovian  deterioration,  while 
a  repair  puts  the  unit  in  the  good  state  by  the  next  decision  time.  Informative  data 
might  become  available  while  producing  without  inspection,  and  inspection  is  not 
edways  perfect.  Hence  the  problem  is  modeled  as  a  partially  observed  Markov  deci¬ 
sion  process  (POMDP).  Furthermore,  we  assume  that  deterioration  of  the  system 
depends  on  an  unknown  parameter,  namely  the  probability  of  the  state  going  from 
the  good  to  the  bad  state  in  one  time  epoch  when  no  repair  is  done.  For  the  case 
of  known  parameters,  we  have  shown  in  our  past  research  that  there  is  an  opti¬ 
mal  policy  for  the  infinite  horizon  average  cost  criterion  that  is  of  the  control  limit 
(bang-bang)  type.  The  adaptive  stochastic  control  problem  is,  however,  much  more 
difficult  than  the  adaptive  estimation  problem,  because  the  presence  of  feedback 
causes  the  system  transitions  to  depend  on  the  parameter  estimates  and  introduces 
discontinuities. 

In  [3]  and  [7],  we  have  analyzed  algorithms  for  this  quality  control  problem  in 
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which  the  parameter  estimates  are  updated  only  after  the  system  is  repaired;  such 
algorithms  are  analogous  to  those  in  which  estimates  in  queueing  systems  are  up¬ 
dated  only  after  each  busy  cycle.  Since  the  system  is  returned  to  the  “good”  state 
after  repair,  one  obtains  a  perfect  observation  of  the  state  at  that  time,  and  our 
algorithm  uses  the  observation  at  the  next  time  to  estimate  the  parameter.  Hence, 
we  develop  parameter  estimation  techniques  based  on  the  information  available  af¬ 
ter  actions  that  reset  the  state  to  a  known  value.  At  these  times,  the  augmented 
state  process  “regenerates,”  its  future  evolution  becoming  independent  of  the  past. 
Using  the  ODE  method,  we  show  that  two  algorithms,  one  based  on  maximum 
likelihood  and  another  based  on  prediction  error,  converge  almost  smrely  to  the  true 
parameter  value.  In  addition,  we  modify  the  method  of  Shwartz  and  Makowski  to 
prove  optimality  of  the  resulting  certainty  equivalent  adaptive  policy,  assuming  only 
the  existence  of  some  sequence  of  parameter  estimates  converging  almost  surely  to 
the  true  parameter  value.  Again,  the  discontinuities  and  partial  observations  in 
this  problem  preclude  the  direct  use  of  previously  existing  methods,  but  we  have 
been  able  to  generalize  the  method  to  problems  such  as  this.  Also,  we  have  avoided 
the  very  strong  standard  assumption  that  the  parameter  estimates  converge  almost 
surely  to  the  true  parameter  value  tuider  any  stationary  policy. 

A  project  in  surveying  the  literature  on  the  ergodic  control  problem  for  discrete¬ 
time  control  Markov  processes  was  completed.  This  was  a  major  effort  which  puts 
together  a  comprehensive  account  of  the  considerable  research  on  this  problem  over 
the  past  three  decades.  Our  exposition  ranges  from  finite  to  Borel  state  and  action 
spaces  and  includes  a  variety  of  methodologies  to  find  and  characterize  optimal 
policies.  We  have  included  a  brief  historical  perspective  of  the  research  efforts  in 
this  area  and  have  compiled  a  substantial  bibliography.  In  the  process  we  have 
identified  several  important  questions  which  are  still  left  open  to  investigation. 

1.1.2.  Stochastic  Control  Switching  Diffusions. 

In  [4]  and  [10]  we  study  a  controlled  switching  diffusion  process  that  arises  in 
numerous  applications  of  systems  with  multiple  modes  or  failure  modes,  including 
the  hierarchical  control  of  flexible  manufacturing  systems.  A  flexible  manufacturing 
system  (FMS)  consists  of  a  set  of  workstations  capable  of  performing  a  number  of 
different  operations  and  interconnected  by  a  transportation  mechanism.  An  FMS 
produces  a  family  of  parts  related  by  similar  operational  requirements  or  by  belong- 
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ing  to  the  same  final  assembly.  The  rapidly  growing  range  of  applicability  of  FMS 
includes  metal  cutting,  assembly  of  printed  circuit  boards,  integrated  circuit  fabrica¬ 
tion,  automobile  assembly  lines,  etc.  The  high  capital  cost  of  an  FMS  demands  very 
efficient  management  of  production  and  maintenance  (repair/replacement)  schedul¬ 
ing  so  that  uncertain  events  such  as  random  demand  fluctuations,  machine  failures, 
inventory  spoilages,  sales  returns,  etc.  can  be  taken  care  of.  The  large  size  of  the 
system  and  its  associated  complexities  make  it  imperative  to  divide  the  control  or 
management  into  a  hierarchy  consisting  of  a  number  of  levels.  Thus,  the  overall 
complex  problem  is  reduced  to  a  number  of  manageable  subproblems  at  each  level, 
and  these  levels  are  linked  by  means  of  a  hierarchical  integrative  system.  It  is  of 
paramount  importance  to  develop  and  study  an  appropriate  mathematical  model 
which  will  facilitate  to  find  on  line  implementable  optimal  feedback  policies. 

We  view  an  FMS  as  a  system  consisting  of  a  number  of  workstations,  with  each 
workstation  having  a  set  of  identical  machines.  A  collection  of  a  number  of  types 
of  different  parts  is  produced.  The  model  we  are  using  involves  a  hybrid  process 
in  continuous  time  whose  state  is  given  by  a  pair  {X{t),  S{t)y  Here,  X{t)  denotes 
the  downstream  buffer  stock  of  parts,  which  may  have  a  negative  value  to  indicate 
a  backlogged  demand.  The  continuous  component  X(t)  is  governed  by  a  controlled 
diffusion  process  with  a  drift  vector  which  depends  on  the  discrete  component  ^(i). 
Thus,  X(t)  switches  from  one  diffusion  path  to  another  as  the  discrete  component 
S{t)  jumps  form  one  state  to  another.  On  the  other  hand,  the  discrete  component 
5(t),  denoting  the  number  of  operational  machines,  is  influenced  by  the  inventory 
size  and  production  scheduling,  and  can  also  be  controlled  by  various  decisions  such 
as  produce,  repair,  replace,  etc.  Hence,  S{t)  evolves  as  a  “controlled  Markov  chain” 
with  a  transition  matrix  depending  on  the  continuous  component. 

This  FMS  model  motivates  the  study  of  a  stochastic  optimization  problem  in  a 
more  abstract  setting  which  is  manifested  in  numerous  other  situations.  For  exam¬ 
ple,  it  is  encoimtered  in  a  hybrid  model  proposed  for  the  study  of  dynamic  phenom¬ 
ena  in  large  scale  interconnected  power  networks,  in  macroeconomic  problems  and 
in  dynamic  renewal  problems  in  general.  Our  treatment  of  the  optimization  prob¬ 
lem  is  based  on  a  convex  analytic  approach  which  is  interesting  in  its  own  right  and 
would  be  more  flexible  and  powerful  for  certain  other  purposes,  e.g.  pathwise  aver¬ 
age  cost  problem  or  problem  with  several  constraints  where  the  analytic  approach 
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does  not  seem  to  be  amenable. 

1.2.  Nonlinear  Systems. 

Most  of  the  research  activity  on  adaptive  control  of  nonlinear  systems  is  still  fo¬ 
cused  on  the  full-state  feedback  case,  although  output-feedback  results  are  beginning 
to  appear.  Adaptive  output-feedback  designs  follow  either  a  direct  model-reference 
path  or  an  indirect  path  via  adaptive  observers.  Current  research  on  adaptive  ob¬ 
servers  for  nonlinear  systems,  indicates  that  the  indirect  path  may  become  promising 
for  adaptive  control.  Continuing  our  work  in  this  direction  we  studied  in  [9]  the 
problem  of  construction  of  observers  for  nonlinear  affine  systems  with  linearizable 
error  dynamics.  Om  aim  was  to  obtain  a  design  for  which  the  diffeomorphism 
transforming  the  nonlinear  system  into  a  nonlinear  observer  form  is  independent  of 
the  choice  of  the  control  input.  Working  with  one- forms  we  have  also  dealt  with 
the  problem  of  constructing  approximate  observers  up  to  a  certain  desired  order. 
This  is  very  important  for  systems  where  the  rather  stringent  conditions  necessary 
for  the  exact  observer  design  are  not  met.  We  have  extended  these  results  to  the 
multiple-input,  multiple-output  case. 

Approximate  linearization  of  nonlinear  systems  becomes  important  for  systems 
where  the  nonlinearities  are  severe  enough  that  exact  linearization  fails.  An  approx¬ 
imate  method  that  linearizes  the  system  up  to  a  certain  order  has  originally  been 
proposed  by  Krener.  Since  less  restrictive  conditions  are  required  for  approximate 
linearization,  this  technique  offers  the  means  of  enlarging  the  class  of  nonlinear  sys¬ 
tems  to  which  linearizing  techniques  are  applicable.  In  [2]  we  study  the  problem 
via  differential  forms.  Despite  the  fact  that  this  is  only  the  dual  to  the  approach 
that  Krener  followed,  we  show  that  it  results  in  substantial  computational  savings. 
Our  method  is  constructive  and  offers  a  simple  solution  to  the  problem. 

In  [1]  we  studied  the  controllability  problem  of  nonlinear  systems  along  closed 
orbits.  The  results  extend  previous  work  in  the  field  on  local  controllability  along 
a  reference  trajectory. 

1.3.  Other  Related  Research. 

We  have  also  made  progress  in  a  number  of  other  related  areas  of  research.  We 
have  started  working  on  a  particular  state  estimation  problem  involving  intercon¬ 
nected  power  systems.  The  objective  here  is  to  develop  a  methodology  for  the 
optimal  placement  of  instrumentation  to  detect  the  location  and  type  of  harmonic 
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sources  in  a  power  network.  Harmonic  distortion  in  distribution  power  systems 
is  reaching  detrimental  levels  and  causes  problems  such  as  overheating  and  failure 
of  equipment,  malfunction  of  protective  equipment,  nuisance  tripping  of  sensitive 
loads,  and  interference  with  commtmication  networks.  Most  of  the  harmonics  sim¬ 
ulation  work  performed  thus  far  has  addressed  harmonic  injection  and  propagation 
in  power  systems,  or,  in  other  words,  the  problem  of  harmonic  power  flow  analysis, 
which  amounts  to  modeling  the  injection  currents  at  harmonic  frequencies  and  then 
calculating  bus  voltages,  line  flows,  and  distortion.  The  converse  of  this  problem,  is 
to  identify  the  injection  currents  at  harmonic  frequencies  based  on  measmrements 
of  selected  bus  voltages  and  line  flows.  The  number  of  instruments  that  can  be 
installed  in  a  network  to  fiu’nish  on-line  measurements  is  limited,  mainly  due  to 
their  high  cost.  The  placement  of  these  devices  becomes,  therefore,  very  critical 
and  constitutes  a  problem  that  has  recently  attracted  much  attention  in  the  scien¬ 
tific  community.  The  problem  described  is  not  confined  to  power  systems— more 
generally,  the  question  addressed  here  is  the  optimal  location  of  sensors  for  state 
estimation.  Our  work  so  far  in  this  problem  is  summarized  in  [6].  We  formulate  the 
problem  in  a  rigorous  mathematical  fashion  and  identify  some  important  inherent 
pathologies.  Further  research  in  this  topic  is  under  way. 

In  the  area  of  discrete  event  dynamical  systems  (BEDS),  we  studied  the  super¬ 
visor  synthesis  problem  through  the  use  of  synchronous  composition  of  the  plant 
and  supervisor,  thus  simplifying  the  BEDS  control  methodology.  In  addition,  we 
derived  optimal  algorithms  for  the  computation  of  supremal  controllable  sublan¬ 
guages.  Corresponding  infinite  horizon  supervisory  control  problems  were  consid¬ 
ered  in  [11].  Stability  and  stabilization  of  BEBS’s  are  studied  in  [12];  these  notions 
are  presented  in  a  more  general  setting  than  in  previous  work.  Efficient  tests  for 
stability  and  stabilizability  are  derived. 
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